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Shannon Meets Carnot: 
Mutual Information Via Thermodynamics 

Ori Shental and Ido Kanter 

Abstract 

In this contribution, the Gaussian channel is represented as an equivalent thermal system allowing to 
express its input-output mutual information in terms of thermodynamic quantities. This thermodynamic 
description of the mutual information is based upon a generalization of the 2 nd thermodynamic law and 
provides an alternative proof to the Guo-Shamai-Verdu theorem, giving an intriguing connection between 
this remarkable theorem and the most fundamental laws of nature - the laws of thermodynamics. 

Index Terms: Thermodynamics, mutual information, Gaussian channel, Guo-Shamai-Verdu 
theorem, minimum mean- square error. 

I. Introduction 

The laws of thermodynamics describe the transport of heat and work in macroscopic (i.e., 
large scale) processes and play a fundamental role in the physical sciences. The theory of 
thermodynamics is primarily an intellectual achievement of the 19 th century. The first analysis 
of heat engines was given by the French engineer Sadi Carnot in his seminal 1824 publication 
('Reflections on the Motive Power of Fire and on Machines Fitted to Develop that Power' [1], 
[2]), laying the foundations to the 2 nd law of thermodynamics. This paper marks the start of 
thermodynamics as a modern science [3]. 
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The classical theory of thermodynamics was formulated in consistent form by giants like 
Joule, Clausius, Lord Kelvin, Gibbs and others. The atomic, or microscopic (i.e., small scale) 
approach to statistical thermodynamics was mainly developed through the pioneering work of 
Clausius, Maxwell, Boltzmann and Gibbs, laying the foundations to the more general discipline 
of statistical mechanics [4]. 

In particular, the 2 nd thermodynamic law, for quasi-static processes, linearly relates the change 
in the entropy, dS, to the amount of heat, dQ, absorbed to a system at equilibrium, dQ = TdS, 
where T is the temperature of the system. However, the 2 nd law, in its classical formulation, is 
suited only for systems with energy Hamiltonian (function), S, which is not an explicit function of 
the temperature and fails to capture the physical behavior of systems with temperature-dependent 
energy levels. While such temperature-dependent energy function is uncommon in the study of 
natural and artificial systems in physics, it surprisingly does arise in modeling communication 
channels, like the popular Gaussian-noise communication channel [5]-[7], as a (quasi-static) 
thermal system [8]— [10]. 

In this contribution, we generalize the 2 nd thermodynamic law to encompass systems with 
temperature-dependent Hamiltonian and obtain the generalized law dQ = TdS+ < d£/dT > dT, 
where < ■ > denotes averaging over the standard Boltzmann distribution. Consequently, it allows 
for an alternative physical description to the Shannon-theoretic notions of information entropy 
and mutual information [11] via the thermodynamic quantities of energy and temperature. As 
an example, the correct expressions for the mutual information of a Gaussian channel with 
Bernoulli- 1/2 input and Gaussian input are computed from the thermodynamic representation, 
where the latter corresponds to the Shannon capacity. 

Guo, Shamai and Verdu [12] have recently revealed a simple, yet powerful relationship 
(hereinafter termed GSV theorem) between information and estimation theories. This cross- 
theory theorem bridges over the notions of Shannon's mutual information and minimum mean- 
squre error (MMSE) for the common additive white Gaussian noise channel. Based on the 
thermodynamic expression of mutual information, the GSV theorem is naturally re-derived. This 
directly links the GSV theorem to the most profound laws of nature and gives the physical origin 
to this remarkable formula. 

The paper is organized as follows. The thermodynamic background is discussed in Sections |n] 
and Unl In Section [IV] the Gaussian channel is represented through an equivalent thermal system 
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giving a thermodynamic expression to the notions of information entropy and mutual information. 
In Section |V] the GSV theorem is proven via thermodynamics. We conclude the paper in 
Section 

We shall use the following notations. P(-) is used to denote either a probability mass function 
(pmf), Pr(-), or a probability density function (pdf), p(-), depending on the random variable 
having either discrete or continuous support, respectively. Random variables are denoted by upper 
case letters and their values denoted by lower case letters. The symbol Ex{-} denotes expectation 
of the random object within the brackets with respect to the subscript random variable. The 
natural logarithm, log, is used. The support of a variable X is denoted by X. 

II. Classical Thermodynamics 

In this section we concisely summarize the fundamental results of classical thermodynamics, 
essential to our discussion. The interested reader is strongly encouraged to find a thorough 
introduction to thermodynamics in one of numerous textbooks {e.g., [13]). 

A thermodynamic system is defined as the part of the universe under consideration, separated 
from the rest of the universe, referred to as environment, surroundings or reservoir, by real or 
imaginary boundary. A non-isolated thermodynamic system can exchange energy in the form of 
heat or work with any other system. Heat is a process by which energy is added to a system from 
a high-temperature source, or lost to a low-temperature sink. Work refers to forms of energy 
transfer which can be accounted for in terms of changes in the macroscopic physical variables 
of the system (e.g., volume or pressure). For example, energy which goes into expanding the 
volume of a system against an external pressure, by driving a piston-head out of a cylinder 
against an external force. Hereinafter we consider a thermal system which does not perform 
work, mechanical or other. This thermal system is assumed to be in equilibrium, that is all the 
descriptive macroscopic parameters of the system are time-independent. We also assume that 
the process of exchanging heat is infinitesimally quasi-static, i.e., it is carried out slowly enough 
that the system remains arbitrarily close to equilibrium at all stages of the process. 

The underlying laws of thermodynamics consist of purely macroscopic statements which make 
no reference to the microscopic properties of the system, i.e., to the molecules or particles of 
which they consist. In the following statements we present the thermodynamic laws in a form 
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relevant to the thermal system under considerationO 
Laws of thermodynamics. 

1 st Law. (conservation of energy) A system in equilibrium can be characterized by a quantity 
U, called the 'internal energy '. If the system is not isolated, interact with another system and 
no work is done by it, the resulting change in the internal energy can be written in the form 

dU = dQ, (1) 

where dQ is the amount of heat absorbed by the system^ 

2 nd Law. (definition of temperature) A system in equilibrium can be characterized by a quantity 
S, called the 'thermodynamic entropy'. If the system is not isolated and undergoes a quasi-static 
infinitesimal process in which it absorbs heat dQ, then 

dS = —, (2) 

where T is a quantity characteristic of the system and is called the 'absolute temperature ' of 
the system. 

3 rd Law. (zero entropy) The thermodynamic entropy S of a system has a limiting property that 

as T — ► 0+, S — ► S'o, (3) 

where Sq is a constant (usually in case of non-degenerate ground state energy) independent 
of all parameters of the particular system. 

Incorporating the three laws of thermodynamics together, a combined law describing the 
thermodynamic entropy as an integration function over the temperature is obtained 

Jo 1 Jo 1 d l Jo 7 

where 

cvm 4 am (5) 

'For conciseness the th law's statement is omitted. 

2 The infinitesimal heat is denoted by d rather than d because, in mathematical terms, it is not an exact differential. The 
integral of an inexact differential depends upon the particular path taken through the space of (thermodynamic) parameters while 
the integral of an exact differential depends only upon the initial and final states. 
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is known as the heat capacity (at constant volume V). The heat capacity is a non-negative 
temperature-dependent measurable quantity describing the amount of heat required to change 
the system's temperature by an infinitesimal degree. 

Let us now define the inverse temperature /3 = {k B T)~ l , where the constant k B is the Boltz- 
mann constant. In this contribution we arbitrarily set k B = 1, thus (3 = 1/T. Hence, the entropy 
integral © can be rewritten in terms of the inverse temperature as 



oc 



Stf) = - I 7^(7)^7, (6) 

where 



Cy{(3) 4 = -T 2 C V (T). (7) 

III. Statistical Thermodynamics 

Moving to the microscopic perspective of statistical thermodynamics [13], [14], the probability, 
P(X = x), of finding the system in any one particular microstate, X = x, of energy level 
£ (X — x) is dictated according to the canonical Boltzmann distribution [4] 

P(X = x) = i exp ( - p£(X = x)) , (8) 

where 

Z^^exp(-^(X = x)) (9) 

is the partition (normalization) function, and the sum extends over all possible microstates of 
the system. 

Applying the Boltzmann distribution, the macroscopic quantities of internal energy and en- 
tropy can be described microscopically as the average energy and the average {— logP(X)}, 
respectively. Explicitly, 

U = E X {S(X)}, (10) 
S = E x {-logP(X)}, (11) 

and it can be easily verified that the following relation holds 

]ogZ = -pU + S. (12) 
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IV. The Gaussian Channel As A Thermal System 

Consider a real-valued channel with input and output random variables X and Y, respectively, 
of the form 

Y = X + N, (13) 

where N ~ A/"(0, 1/snr) is a Gaussian noise independent of X, and snr > is the channel's 
signal-to-noise ratio (SNR). The input is taken from a probability distribution P(X) that satisfies 
E X {X 2 } < oo|] 

The Gaussian channel ((131 (and any other communication channel) can be also viewed as a 
physical system, operating under the laws of thermodynamics. The microstates of the thermal 
system are equivalent to the hidden values of the channel's input X. A comparison of the 
channel's a-posteriori probability distribution, given by 

= P(* = ,y = y |* = J ) 

p(Y = y) 
/ snFP(X = x) ( snr / 



exp - —(y- x y) (15) 



2tt P {Y = y) V 2 
exp ( - snr(-*y + f - ^^)) 



(16) 



E^ex P (-snr(-xy + f-^^))' 
with the Boltzmann distribution law © yields the following mapping of the inverse temperature 
and energy of the equivalent thermal system 

snr -> /3, (17) 
-^ + y- lQgP( ^ = 3:) - S(X = x\Y = y;0). (18) 



Note that the temperature (i.e., the noise variance according to the mapping (fTTI) ) can be 
increased gradually from the absolute zero to its target value T — 1/(5 in infinitesimally small 
steps, thus the Gaussian channel system can remain arbitrarily close to equilibrium at all stages 
of this process. Hence, the equivalent thermal system exhibits a quasi-static infinitesimal process. 
Interestingly, the notion of quasi-statics is reminiscent to the concept of Gaussian pipe in the 
SNR- incremental Gaussian channel approach taken by Guo et ah [12, Section II-C]. Thus, we 

Similarly to the formulation in [12] for E X {X 2 } = 1, snr follows the usual notion of SNR. For E X {X 2 } 1, snr can be 
regarded as the gain in the output SNR due to the channel. 
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can apply the entropy integral © obtained from thermodynamics to the thermal system being 
equivalent to the Gaussian channel, yielding 



S(J3) = S{X\Y = y;0) = -J 1 C v ( 1 )d 1 =-J 7 ^^-^d 7 , (19) 

where following (flOl ) the internal energy, U(Y — y; (5) = ~E x \y{£(X\Y = y)}, is the energy 
averaged over all possible values of X, given y. 

The posterior information (Shannon) entropy, H(X\Y; f3), (in nats) of the channel can be 
expressed via the thermodynamic entropy conditioned on Y = y, S(X\Y = y; (3) (fl9l) . as 

H(X\Y: >') = Ev{,S'f..V(y = //: >)} = -E Y { / °° 7) ^ , (2 (!) 



P 



The input's entropy can also be reformulated in a similar manner, since H(X) = H(X\Y; (3 = 0). 
Hence, 



oo 



H(X) = -E Y { I 1 d ^p±d 1 }. (21) 







Now, the input-output mutual information can be described via thermodynamic quantities, 
namely the energy, £ , and inverse temperature, (3, as 



-M l\^f^-dj} (23) 



I(X;Y) = I{/3)±H(X)-H(X\Y;P) (22) 

= -[ 7 E y {?7(y;7)}]J+Ey|^^;7)d7|, (24) 

where (1241) is obtained using integration by parts. 

Note that this thermodynamic interpretation to the mutual information holds not only for the 
Gaussian channel, but also for any channel which can be described by a thermal system exhibiting 
quasi-static heat transfer. In the following we illustrate the utilization of (|24l) by re-deriving the 
correct expression for the mutual information of a Gaussian channel with Bernoulli- 1/2 input. 

Example: Gaussian Channel with Bernoulli-1/2 Input 

Since the input X in this case is binary and equiprobable, i.e. P(X = 1) = P(X = — 1) = 
1/2, the X 2 /2 = 1/2 and logP(X = x)/(3 terms of the Gaussian channel's energy (TT8T) are 
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independent of X and can be droppedj, leaving us with the expression, independent of ft, 

£{X = x\Y = y) = -xy. (25) 

The a-posteriori probability mass function gets the form 

Pr(X = x\Y = y)= ( ^^ V \ R y x 6 ±1. (26) 
exp (fjy) + exp {-(3y) 

Hence, the internal energy is 

U(Y = y;P)= E xlY {£(X\Y = y)} = -y tanh(/3y). (27) 

The marginal pdf of the output is then given by 

VP 



p(Y = y) 



2V2tt 
Thus, 



(exp ( - |(y - l) 2 ) + exp ( - |(y + l) 2 ) j . (28) 



/oo 
ytanh(/fy)p(y = y)dy (29) 
-co 

= -^j= i^exp ( - |(y - l) 2 ) - exp ( - |(y + l) 2 ) jd*30) 
= f (1 - (-1)) = /3 (31) 



and 



fj3 I poo p/3 

E y <J I U(Y; 7 )d-y} = / p(Y = y) U(Y = y ]1 )d 1 dy (32) 



J -oo JO 



i f°° ( y 2 \ 

giving, based on (1241) . 

J(/3) ={3--^=J exp T - ^ log cosh (/3 - ^A/)*/, (34) 

which is identical to the known Shannon-theoretic result (see, e.g., [12, eq. (18)] and [15, p. 
274]). □ 

4 Cancelled out by the same terms from the denominator in J16t . 
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A. Generalized 2 nd Law 

When trying to repeat this exercise for a Gaussian input one finds that (1241) fails to reproduce 
/(snr) = l/21og(l + snr), which is the celebrated formula for the Shannon capacity of the 
Gaussian channel. This observation can be explained as follows. The 2 nd thermodynamic law, 
as stated above, holds only for systems with an energy function, £ , which is not a function 
of the temperature. While the temperature-independence of £ is a well-known conception in 
the investigation of both natural and artificial systems in thermodynamics, and physics at large, 
such independence does not necessarily hold for communication channels and particularly for 
the Gaussian channel. Actually, one can easily observe that the Gaussian channel's energy (TT8T) 
is indeed an explicit function of the temperature via the logP(X = x)/j3 term, unless this 
term can be dropped by absorbing it into the partition function, as happens for equiprobable 
input distributions, like the discrete Bernoulli- 1/2 or the continuous uniform distributions. This 
is exactly the reason why ((24]) does succeed to compute correctly the mutual information for 
the particular case of a Bernoulli- 1/2 input source. 

In order to capture the temperature-dependent nature of the energy in systems like the commu- 
nication channel, we generalize the formulation of the 2 nd law of thermodynamics dS = dQ/T. 

Generalized 2 nd Law. (redefinition of temperature) If the thermal system is not isolated and 
undergoes a quasi-static infinitesimal process in which it absorbs heat dQ, then 

T T \ dT ) v ' 

Proof: The differential of the partition function's logarithm, log Z, can be written as 

dlogZ=^^dp. (36) 

dp 

Utilizing the identity (fT2l) . one obtains 

dS = d{pu) + -| dp. (37) 

(XL) 

Since for T-dependent energy 
we get 

dS = d(/3U)- Udp - [3E X | \dp = pdU - pE x j J dp, (39) 
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where the r.h.s. results from Leibniz's law (product rule). Recalling that according to the I s * law 
dU = dQ concludes the proof. ■ 

The generalized 2 nd law of thermodynamics (1351) has a clear physical interpretation. For 
simplicity, let us assume that the examined system is characterized by a comb of discrete energy 
levels SI, S2, .... The heat absorbed into the T-dependent system has the following dual effect: 
A first contribution of the heat, dU — Ex{dS(X) / dT}dT , increases the temperature of the 
system while the second contribution, F,x{dS(X)/dT}dT, goes for shifting the energy comb. 
However, the shift of the energy comb does not affect the entropy, since the occupation of 
each energy level remains the same, and the entropy is independent of the energy values which 
stand behind the labels SI, S2, . . .. The change in the entropy can be done only by moving part 
of the occupation of one tooth of the energy comb to the neighboring teeth, and this can be 
achieved only by changing the temperature. Hence, the effective heat contributing to the entropy 
is dQ — ~Ex{dS(X)/dT}dT, and this is the physical explanation to the generalized 2 nd law (|35l) . 
Note that for T-independent energy, the classical 2 nd law © is immediately obtained. 

Note also, that the I s * law remains unaffected, dU = dQ, since both ways of heat flow 
absorption into the system are eventually contributing to the average internal energy U . The 
generalized 2 nd law specifies the trajectory, the weight of each one of the two possible heat 
flows, at a given temperature. 

The temperature, originally defined by the 2 nd law as T = dQ/dS, is redefined now as 

1 dS/dQ dS/dQ 



rp 1 _ V x {dS(X)/dT} 1 _ V x {d£(X)/dT} 
1 dQ/dT 1 CV(T) 



(40) 



This redefinition has a more complex form and involves an implicit function of T since the 
temperature appears on both sides of the equation. 

Based on the generalized 2 nd law, the thermodynamic expression for the mutual informa- 
tion (|2~2|) of quasi-static (e.g., Gaussian) communication channels can be reformulated as 

I(X;Y) = -E Y | j\(dU(Y; 7 ) + E X]Y j T) }d 7 ) j (41) 




-[ T E y {[/(Y; T )}]^ + E y | jf (u(Y; 1 ) + 1 E xlY ^ dS{X J^ l) |)d 7 1(43) 
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Again, the example of a Gaussian channel with, this time, standard Gaussian input is used to 
illustrate the utilization of (|43l) for the correct derivation of the mutual information. 

Example: Gaussian Channel with 7V(0, 1) Input 

Since in this case log(P(X))//3 = —x 2 /(2(3), the energy (fTBl of the Gaussian channel system 
becomes an explicit function of (3, given by 




(44) 



and the derivative of this function with respect to (3 yields 

8S(X = x\Y = y;P) 

The a-posteriori probability density function is 




(45) 




(46) 



Hence, the internal energy is 



U(Y = y-(3) 



E xlY {S(X\Y 



y,(3)} 



y 2 (3 J_ 
2(1 + /?) + 2/3' 



(47) 



and the derivative of the energy averaged over all possible inputs is 




(48) 



The marginal pdf of the output is given by 




(49) 



Thus, 




2 2 




(50) 
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and 

rP 



E 



Y ' 



E 



£(u(Y„) +i ^{*y^})^ (51) 

l + 7 l + 7 i y 2 Y 2 

+ — - log(l + + - log(l + 7) - M 1 + 7) 



2 2 7 &v u 2 &v "2 &v u 2(1 + 7) 



- 






1(52) 


- 








+ 







8 1 + 3 11 F 2 F 2 

Ev<! - ? + ^log(l + /?)-2 + 2 1 °g(l + /5)- T log(l+/5)-^ rT 

(53) 

f + I]og(l + /3) (54) 



2 2 
giving, based on (|43l 



/(X ; r) = ^log(l + /3) (55) 



and the Shannon capacity [11] is derived from the perspective of thermodynamics. 

V. Guo-Shamai-Verdu Theorem 

In this section we prove the Guo-Shamai-Verdu (GSV) theorem from the 2 nd law of thermo- 
dynamics for systems with T-dependent energy and the resulting thermodynamic representation 
of the mutual information. Thus, we show that this fascinating relation between information 
theory and estimation theory is actually an evolution of the most profound laws of nature. 

To start with, let us restate the GSV theorem. Consider a Gaussian channel of the form 

Y = VsnrX + N, (56) 

where N ~ A/"(0, 1) is a standard Gaussian noise independent of X. The mutual information, 
/(snr) ((22)) . and the minimum mean-square error, defined as 

mmse(X| v / snrX + N) = mmse(snr) = E XtY {(X - E xlY {X\Y; snr}) 2 }, (57) 

are both a function of snr and maintain the following relation. 

GSV Theorem. [12, Theorem 1 ] For every input distribution P(X) that satisfies Kx{X 2 } < oo, 

-I(X; y/srirX + N) = -mmse(X\VsnrX + N). (58) 



dsnr v '2 
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Note that the GSV-based expression of the mutual information, 

, 1 f snr 

I(X; y/sfifX + N) = - / mmse(X\y/iX + N)dy, (59) 

2 Jo 

resembles its thermodynamic expression (1421) in the sense that both are an outcome of integration 
with respect to SNR (or inverse temperature, [3). Hence, the roots of this integration in the GSV 
theorem may be attributed to the 2 nd law of thermodynamics. Note however to the opposite order 
of integration in the two expressions, where in the GSV expression (|59l) the inner integration 
(within the definition of MMSE) is over y and the outer integration is over SNR, and vice versa 
for the thermodynamic expression (l42~l) . Exchanging the order of integration in the latter (which 
is not trivial since p(Y = y) is itself a function of (3) yields, as we shall see, an integrand of d(3 
which is equal to mmse(/3)/2. In the following proof Lemma 1 from [12], which underlines the 
main proof of the GSV theorem in [12], is proven directly from the thermodynamic description 
of the mutual information (|4T|) . 

Proof: Adopting the SNR-incremental channel approach [12, Eq. (30)-(41)] and mapping 
again snr — > (3 

dI(X;Y) I(X;n)-/(*;Y 2 ) I(/3 + S) - I(J3) I(X;Yi|Y 2 ) 



dp 5 5 6 

where I(X; Yi|Y 2 ) is the mutual information of the incremental Gaussian channel 



(60) 



Y l = V6X + N, (61) 

where N is a standard Gaussian noise, X is taken from the conditional probability P(X\Y2), 
5 -> and 

Y 2 =X+Af{0,l/P) (62) 



with X ~ P{X). Hence, we have to prove 

I(X;Yi|Y 2 ) 1 



mmse(/3). (63) 



5 2 

Now, the principles of thermodynamics come into action. For this incremental channel (l6"TT) . the 
energy and its derivative are given by 

S {X = x\Y x = yx,Y 2 = y 2 ;5,/3) = --j= + — (64) 



and 



d c(y iv v x a\ % y i5 2 , 1 °SP{X = x\Y 2 =y 2 ;f3) 
— S {X = x\Yx = y l7 Y 2 = y 2 ;5,(3) = — - — + — . (65) 
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Using the thermodynamic expression for the mutual information (I4TT) and recalling that § — > 0, 
one gets 

I(X;Y 1 \Y 2 = y 2 ) = -E Yi1 yA ^7(^(^2 = V2] 7, P) + ®x\y u y, ^S(X\Y x , Y 2 = y 2 ; 7, /?)}d 7 ) 



-E 



(66) 

ttl* |E X | yii y 2 |^(X|n, ^2 = V2-, S, (3) + 5 2 j- 5 £{X\Y u Y 2 = y 2 ; S, (3) J J (67) 

w / E X | yi , ya {x|y 1 ,y 2 = y 2 }y lV ^ ^E X | yi , y2 {x 2 |r 1; r 2 = y2 > \ 

-E Yl | Y2 <^ + \ (68) 

-E yi |y 2 | - y^Y 1 E xlY2 {X\Y 2 = y 2 } + ^E X |y 2 {X 2 |F 2 = y 2 }| (69) 

- ( - E 2 xlY2 {X\Y 2 = y 2 } + E x ,y 2 {X 2 |F 2 = y 2 }J (70) 

E x ,y 2 {(X - E x ,y 2 {X|y 2 = y 2 }f\Y 2 = y 2 }, (71) 



2 

where (l69l is based on the fact that for an infinitesimal snr, 5^0, the expectation E^y^y, — > E x \y 2 
while ([70]) results from the independence of N (loTI) with Y" 2 (1621) [12, Section II-C]. Averaging 
over Y 2 on both sides of the equation, we obtain the desired result 

/(X; Y ± \Y 2 ) = ^E x ,y 2 {(X - E x ,y 2 {X|r 2 }) 2 } = ^mmse(/3). (72) 



VI. CONCLUSION 

In this paper, the mutual information of Gaussian channels is described via thermodynamic 
terminology. As a byproduct, an intimate link is revealed between the GSV theorem and the 
basic laws of thermodynamics. More generally, the revised 2 nd law for thermal systems with 
temperature-dependent energy levels enables to quantitatively bridge between the realm of ther- 
modynamics and information theory for quasi-static systems. It is anticipated that this substantial 
theoretical connection between the foundation laws of thermodynamics and information theory 
will open a horizon for new discoveries and development in the study of both artificial and 
natural systems, towards a possibly more synergetic foundation to these two vital disciplines. 
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